Comprehensive Cross-Hierarchy Cluster Agreement Evaluation

Hierarchical clustering represents a family of widely used clustering approaches that can organize objects into a hierarchy based on the similarity in objects’ feature values. One significant obstacle facing hierarchical clustering research today is the lack of general and robust evaluation methods. Existing works rely on a range of evaluation techniques including both internal (no ground-truth is considered in evaluation) and external measures (results are compared to ground-truth semantic structure). The existing internal techniques may have strong hierarchical validity, but the available external measures were not developed specifically for hierarchies. This lack of specificity prevents them from comparing hierarchy structures in a holistic, principled way. To address this problem, we propose the Hierarchy Agreement Index, a novel hierarchy comparison algorithm that holistically compares two hierarchical clustering results (e.g. ground-truth and automatically learned hierarchies) and rates their structural agreement on a 0-1 scale. We compare the proposed evaluation method with a baseline approach (based on computing F-Score results between individual nodes in the two hierarchies) on a set of unsupervised and semi-supervised hierarchical clustering results, and observe that the proposed Hierarchy Agreement Index provides more intuitive and reasonable evaluation of the learned hierarchies. Introduction and Related Work Research into flat clustering methods benefits greatly from the availability of powerful cluster evaluation tools, ranging from the original Rand Index (Rand 1971) to more modern methods such as V-Measure (Rosenberg and Hirschberg 2007), that allow researches to effectively judge their clustering results against some ground-truth objective, and thus compare the relative performance of different methods. Unfortunately, hierarchical clustering does not enjoy such a wealth of viable, established evaluation techniques. While there do exist well-grounded internal (i.e. with no ground truth information) measures of hierarchy quality (Eriksson et al. 2011), researchers interested on measuring the external, semantic meaning of a hierarchy are forced to resort to a wide array of questionable and limited methods. Some, for instance, simply test their methods on non-hierarchical data, Copyright c © 2013, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. and cut their hierarchies to produce a flat segmentation for evaluation (Zheng and Li 2011; Kim and Lee 2000). Currently, the most common external evaluation technique is a method we refer to as Cluster F-Measure (CFM). It was originally proposed in (Larsen and Aone 1999) and works (on either flat or hierarchical clusterings) by matching each ground-truth cluster/hierarchy node to the “best” test cluster/hierarchy node (where match quality is determined by F (c1, c2) = 2·P ·R P+R : the F-score for cluster c2 on cluster c1). The overall score, then, is just the cluster-size-weighted average of the best scores. This method is applicable to hierarchies, but not specific to them, and when applied to a hierarchical clustering solution completely ignores the actual structure of the hierarchy, treating each node as an unrelated flat cluster. We would argue that a strong tool for evaluating hierarchical clustering solutions must take account of the hierarchical relationships between nodes and elements, and that the CFM approach is thus inadequate. We instead propose a novel evaluation technique that directly encodes and compares the entire structure of two hierarchies. We refer to our method, which can be thought of as an extension of the classical Rand Index to the hierarchical case, as the Hierarchy Agreement Index (HAI). Hierarchy Agreement Index In both the Rand Index and our proposed HAI, the total comparison score is computed via: S(C1, C2) = 1− 1 N2 N ∑